You can find the free to use application at https://protected-mountain-28715.herokuapp.com.

Author: Nikolaos Toumpanakis

Email: toump.nick@gmail.com GitHub: https://github.com/ntoump/ Youtube: https://www.youtube.com/channel/UCFgF1lHh0fRQxY9CqpyBZgw

Training Walkthrough

I should mention that, in this Walkthrough, as well as in the deployed application (https://protected-mountain-28715.herokuapp.com), I use Multi-Label Classification alongside one-hot encoding, so that each image is not limited to one category as output. Instead, the theoritical maximum output is all categories on which the model has been trained (although that is extremely unlikely), and the theoritical minimum is none of these categories (as the model may not be confident enough in any of its predictions). In this last case, as it's highlighted in the Specification of the application, the model will output an error message.

Data

df = pd.read_csv('CSVs/only_images4.csv')
df
fname labels
0 C001.jpg Cloth
1 C002.jpg Cloth
2 C003.jpg Cloth
3 C004.jpg Cloth
4 C005.jpg Cloth
... ... ...
1011 No094.jpg No_mask!
1012 No096.jpg No_mask!
1013 No097.jpg No_mask!
1014 No099.jpg No_mask!
1015 No100.jpg No_mask!

1016 rows × 2 columns

First try

This is my first try of the "masksv3" (i.e. the 3rd Version of this application) at creating an ever better model by using Deep Convolutional Neural Networks, continuing exactly where "masksv2" left off.

In comparison to the previous Version, in this first try I only changed the dataset, filtering out many mislabeled images and adding a few, as well as customizing my previous dataset to support one-hot encoding. At the same time, everything else (e.g. the hyperparameters) is left as is.

After reviewing the final accuracy of this first model, I'll try to improve on that by tweaking the hyperparameters.

Wish me luck!

# Get dblock, dsets, and dls
# Notice that no Presizing is used (all credits to fastai for their incredible work) in this first DataBlock
dblock = DataBlock(blocks=(ImageBlock, MultiCategoryBlock),
                   splitter=RandomSplitter(),
                   get_x=get_x, 
                   get_y=get_y,
                   item_tfms = RandomResizedCrop(128, min_scale=0.35))

dsets = dblock.datasets(df)
dls = dblock.dataloaders(df)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-1-b0a99ccd95eb> in <module>
      1 # Get dblock, dsets, and dls
      2 # Notice that no Presizing is used (all credits to fastai)
----> 3 dblock = DataBlock(blocks=(ImageBlock, MultiCategoryBlock),
      4                    splitter=RandomSplitter(),
      5                    get_x=get_x,

NameError: name 'DataBlock' is not defined
# Everything looks good
dls.show_batch()

Training

I start off with a small threshold (0.2), ResNet50, and a semi-random base learning rate. Furthermore, I train using _finetune, because it could be a good starting point to begin to understand what the differences are (in practical terms and applications) between the fastai model training tools (including fit, _fit_onecycle, and _finetune).

First set of epochs

learn = cnn_learner(dls, resnet50, metrics=partial(accuracy_multi, thresh=0.2))
learn.fine_tune(3, base_lr=3e-3, freeze_epochs=4)
epoch train_loss valid_loss accuracy_multi time
0 0.920142 0.598956 0.413793 00:03
1 0.776364 0.499953 0.561576 00:03
2 0.672306 0.435738 0.629310 00:03
3 0.583970 0.318443 0.676724 00:03
epoch train_loss valid_loss accuracy_multi time
0 0.314077 0.264383 0.758005 00:03
1 0.261582 0.194348 0.862685 00:03
2 0.220241 0.161082 0.887315 00:03

The accuracy seems to have improved greatly in a few seconds (because of using a P5000 NVIDIA GPU) and epochs of training, which indicates that there was plenty of room to grow

Next, I sketch the lr to loss relationship, and get the suggested lrs, in order to train even further with a better informed lr guess.

learn.lr_find()
SuggestedLRs(lr_min=0.0003019951749593019, lr_steep=0.0006918309954926372)

There does not seem to be a clear slope downwards, that's why I'll just use the lr_min.

Second set of epochs

learn.fine_tune(3, base_lr=0.0003019951749593019, freeze_epochs=4)
epoch train_loss valid_loss accuracy_multi time
0 0.169711 0.157992 0.887931 00:03
1 0.168780 0.148448 0.906404 00:03
2 0.160508 0.140320 0.914409 00:03
3 0.153258 0.132069 0.927340 00:03
epoch train_loss valid_loss accuracy_multi time
0 0.133067 0.133076 0.929803 00:03
1 0.130006 0.130573 0.934729 00:03
2 0.129661 0.129405 0.937192 00:03

The accuracy has skyrocketed to almost 94%. Thus far, I can say that we've done pretty well.

lr_min, lr_steep = learn.lr_find()

Let's train some more.

Third and final set of epochs

learn.fine_tune(2, base_lr=lr_min)
epoch train_loss valid_loss accuracy_multi time
0 0.125494 0.127268 0.939655 00:03
epoch train_loss valid_loss accuracy_multi time
0 0.112003 0.122138 0.944581 00:03
1 0.111319 0.120099 0.943965 00:03
lr_min, lr_steep = learn.lr_find()

Not bad! We just saw, for the first time, a decrease in accuracy (something that I was expecting, that's why I trained it only a few epochs per set). That's not necessarily a sign of overfitting. It would be a good opportunity, though, to get the predictions of the model, and plot the threshold to accuracy relationship.

preds,targs = learn.get_preds()
xs = torch.linspace(0.05,0.95,29)
accs = [accuracy_multi(preds, targs, thresh=i, sigmoid=False) for i in xs]
plt.plot(xs,accs);

The above plot shows that there is a positive relationship between threshold and accuracy. Therefore, the higher is the chosen threshold, the higher is expected to be the accuracy of the model. It is also important to note that there isn't such a great difference between choosing, say, 0.3 and 0.85 as your threshold, perhaps not even a percentage point.

First try - Review

I am fairly impressed by the model's performance. Reaching 95% accuracy, considering the variation of the ds *, the limited training time (& costs), and the uncustomized hyperparameter choices, is almost excellent.

Now onwards to manual testing.

I have a few dozen images, against which I will check my model's accuracy. These are the ones used for evaluating the masksv2 application's model as well.

*: The dataset contains, at this point, seven (#7+1) distinct categories, namely: (#7) ['-','Cloth','Gas','KN95','Medical','No_mask!','Shield', 'Surgical'].

Each category refers to a type of face mask, except for "-" and "No_mask!". The last one suggests that there is a maskless face in the given image, whereas the first one ("-") is part of an experiment I conduct named "Intended Noise" (see below).

Manual Testing

I have three folders of images for my manual testing: "NEW", "NEU", and all the other pictures you see below.

[The names of the images are supposed to be related to their content, while that is not always necessarily the case.]

ls
NEU/  c.jpg   c1.jpg  c3.jpg  f.jpg    ff0.jpg  kn.jfif   kn1.jfif  kn3.jfif
NEW/  c0.jpg  c2.jpg  c4.jpg  ff.jfif  k.jfif   kn0.jfif  kn2.jpg   m.jpg
# Built a quick loop to do the work for me
# Below you can see the predictions of the model

import os

pather = Path('/notebooks/Manual')

for img in os.listdir(pather):
    if not img[0] == '.':
        if not img[0].isupper():
            print(img, learn.predict(img))
c.jpg ((#1) ['Surgical'], tensor([False, False, False, False, False, False, False,  True]), tensor([0.0775, 0.1155, 0.0063, 0.0130, 0.0673, 0.0341, 0.2313, 0.9998]))
c0.jpg ((#2) ['Cloth','Surgical'], tensor([False,  True, False, False, False, False, False,  True]), tensor([0.0728, 0.9934, 0.0024, 0.0257, 0.2283, 0.0275, 0.0254, 0.7986]))
c1.jpg ((#1) ['Cloth'], tensor([False,  True, False, False, False, False, False, False]), tensor([0.1199, 0.9999, 0.0045, 0.0559, 0.1601, 0.0538, 0.0173, 0.0220]))
c2.jpg ((#2) ['Cloth','Surgical'], tensor([False,  True, False, False, False, False, False,  True]), tensor([0.0886, 0.9984, 0.0017, 0.0033, 0.1045, 0.0977, 0.2173, 0.6740]))
c3.jpg ((#1) ['Cloth'], tensor([False,  True, False, False, False, False, False, False]), tensor([0.0635, 1.0000, 0.0062, 0.0124, 0.1405, 0.0456, 0.0129, 0.0107]))
c4.jpg ((#1) ['KN95'], tensor([False, False, False,  True, False, False, False, False]), tensor([0.0337, 0.4820, 0.0034, 0.9528, 0.1218, 0.0256, 0.0292, 0.2378]))
f.jpg ((#1) ['Shield'], tensor([False, False, False, False, False, False,  True, False]), tensor([9.1857e-02, 1.9036e-02, 7.7392e-04, 4.6145e-03, 9.7188e-02, 8.0676e-03, 1.0000e+00, 1.9384e-01]))
ff.jfif ((#2) ['Gas','KN95'], tensor([False, False,  True,  True, False, False, False, False]), tensor([0.0387, 0.0341, 0.9998, 0.7817, 0.1651, 0.0845, 0.0285, 0.0202]))
ff0.jpg ((#2) ['Shield','Surgical'], tensor([False, False, False, False, False, False,  True,  True]), tensor([0.0545, 0.0850, 0.0047, 0.1072, 0.2291, 0.0347, 0.9999, 1.0000]))
k.jfif ((#2) ['Cloth','Shield'], tensor([False,  True, False, False, False, False,  True, False]), tensor([0.0819, 0.9689, 0.0013, 0.1041, 0.1234, 0.0768, 0.6258, 0.0066]))
kn.jfif ((#1) ['KN95'], tensor([False, False, False,  True, False, False, False, False]), tensor([0.0695, 0.0832, 0.0131, 0.5997, 0.2483, 0.0758, 0.0988, 0.1063]))
kn0.jfif ((#1) ['KN95'], tensor([False, False, False,  True, False, False, False, False]), tensor([0.0559, 0.1427, 0.0032, 0.9986, 0.1310, 0.0318, 0.0668, 0.0511]))
kn1.jfif ((#1) ['KN95'], tensor([False, False, False,  True, False, False, False, False]), tensor([0.1210, 0.0280, 0.0057, 1.0000, 0.1297, 0.0482, 0.0182, 0.0055]))
kn2.jpg ((#2) ['KN95','Surgical'], tensor([False, False, False,  True, False, False, False,  True]), tensor([0.0345, 0.0628, 0.0126, 0.5065, 0.1243, 0.0679, 0.0438, 0.9757]))
kn3.jfif ((#1) ['KN95'], tensor([False, False, False,  True, False, False, False, False]), tensor([0.0193, 0.0180, 0.0032, 1.0000, 0.1066, 0.0293, 0.0219, 0.0073]))
m.jpg ((#1) ['Surgical'], tensor([False, False, False, False, False, False, False,  True]), tensor([0.0604, 0.0252, 0.0092, 0.0068, 0.1327, 0.1073, 0.0532, 1.0000]))
for img in os.listdir(pather/'NEW'):
    print(img, learn.predict(img))
0.jpg ((#1) ['Cloth'], tensor([False,  True, False, False, False, False, False, False]), tensor([0.2064, 0.7891, 0.0167, 0.0831, 0.1360, 0.0263, 0.1972, 0.0074]))
1.jpg ((#1) ['Medical'], tensor([False, False, False, False,  True, False, False, False]), tensor([0.0397, 0.0667, 0.0052, 0.0510, 0.8644, 0.0356, 0.0485, 0.0339]))
10.jpg ((#2) ['Cloth','Surgical'], tensor([False,  True, False, False, False, False, False,  True]), tensor([0.0320, 0.9658, 0.0103, 0.1079, 0.3641, 0.0235, 0.0414, 0.6926]))
100.jpg ((#2) ['Cloth','Medical'], tensor([False,  True, False, False,  True, False, False, False]), tensor([0.0284, 0.9688, 0.0715, 0.4539, 0.9123, 0.2135, 0.0244, 0.0144]))
3.jpg ((#1) ['Medical'], tensor([False, False, False, False,  True, False, False, False]), tensor([0.0089, 0.1283, 0.0151, 0.3042, 0.9984, 0.1028, 0.0139, 0.0350]))
30.jpg ((#3) ['Cloth','KN95','Medical'], tensor([False,  True, False,  True,  True, False, False, False]), tensor([0.0420, 0.9991, 0.0048, 0.6708, 0.9817, 0.0964, 0.0404, 0.0023]))
for img in os.listdir(pather/'NEU'):
    print(img, learn.predict(img))
f.jpg ((#1) ['No_mask!'], tensor([False, False, False, False, False,  True, False, False]), tensor([0.0688, 0.0212, 0.0060, 0.0261, 0.1245, 1.0000, 0.0367, 0.0180]))
f0.jpg ((#1) ['No_mask!'], tensor([False, False, False, False, False,  True, False, False]), tensor([0.0323, 0.0214, 0.0043, 0.0135, 0.0879, 1.0000, 0.0364, 0.0129]))
f1.jpg ((#1) ['No_mask!'], tensor([False, False, False, False, False,  True, False, False]), tensor([0.0420, 0.0315, 0.0054, 0.0091, 0.0844, 1.0000, 0.0286, 0.0094]))
f2.jpg ((#1) ['No_mask!'], tensor([False, False, False, False, False,  True, False, False]), tensor([0.0301, 0.0200, 0.0069, 0.0065, 0.1041, 1.0000, 0.0287, 0.0204]))
f3.jpg ((#1) ['No_mask!'], tensor([False, False, False, False, False,  True, False, False]), tensor([0.0465, 0.0427, 0.0093, 0.0152, 0.1351, 1.0000, 0.0774, 0.0113]))
f4.jpg ((#1) ['No_mask!'], tensor([False, False, False, False, False,  True, False, False]), tensor([0.0402, 0.0260, 0.0057, 0.0118, 0.0923, 1.0000, 0.0645, 0.0311]))
f5.jpg ((#1) ['No_mask!'], tensor([False, False, False, False, False,  True, False, False]), tensor([0.0531, 0.0342, 0.0030, 0.0058, 0.1081, 1.0000, 0.0260, 0.0279]))
i0.jpg ((#1) ['-'], tensor([ True, False, False, False, False, False, False, False]), tensor([0.9999, 0.0137, 0.0048, 0.0347, 0.0524, 0.0716, 0.0443, 0.0584]))
i1.jpg ((#1) ['-'], tensor([ True, False, False, False, False, False, False, False]), tensor([0.9998, 0.2033, 0.0035, 0.0099, 0.0363, 0.0421, 0.0947, 0.0159]))
i2.jpg ((#2) ['-','Shield'], tensor([ True, False, False, False, False, False,  True, False]), tensor([0.9941, 0.1205, 0.0118, 0.0185, 0.1548, 0.0936, 0.5295, 0.0373]))
i3.jpg ((#1) ['-'], tensor([ True, False, False, False, False, False, False, False]), tensor([1.0000, 0.0770, 0.0095, 0.0109, 0.1565, 0.0578, 0.0114, 0.0063]))
i4.jpg ((#1) ['-'], tensor([ True, False, False, False, False, False, False, False]), tensor([1.0000, 0.0118, 0.0027, 0.0067, 0.1234, 0.1374, 0.0240, 0.0239]))
l0.jpg ((#1) ['-'], tensor([ True, False, False, False, False, False, False, False]), tensor([1.0000, 0.0766, 0.0029, 0.0092, 0.0801, 0.0751, 0.0630, 0.0171]))
l1.jpg ((#2) ['-','Cloth'], tensor([ True,  True, False, False, False, False, False, False]), tensor([1.0000, 0.8937, 0.0038, 0.1679, 0.3973, 0.0362, 0.0091, 0.0034]))
l2.jpg ((#1) ['-'], tensor([ True, False, False, False, False, False, False, False]), tensor([1.0000, 0.1106, 0.0068, 0.0163, 0.1903, 0.0247, 0.0390, 0.0029]))
l3.jpg ((#1) ['-'], tensor([ True, False, False, False, False, False, False, False]), tensor([1.0000, 0.0251, 0.0066, 0.0144, 0.1485, 0.0422, 0.0181, 0.0456]))
l4.jpg ((#1) ['-'], tensor([ True, False, False, False, False, False, False, False]), tensor([1.0000, 0.2441, 0.0053, 0.0287, 0.0584, 0.0346, 0.0988, 0.0175]))

Manual Check - Review

Spare images

The model did not do particularly well with the spare images. I'll get back to that very soon.

"NEW"

The "NEW" folder contains images with many people wearing masks, rather than just one person at a time (the dataset contained only such images at this point; with one (mostly) mask per image). It was at this point, while reviewing these results, that I realized my blunder; I had included two categories in the dataset for the same type of masks (namely, "Surgical" and "Medical").

This was obviously done by accident, as I had not found a comfortable API to clean the data (the fastai one does not work for multi-label classification), therefore I had to do it by hand. That gave me, nontheless, an idea; what if I included all "Surgical" images with the "Medical" labeled ones into a common category, named "Medical" (To be continued...)?

"NEU"

Now that's where the model really shined. You see, the images in this category are part of what I call "Intended Noise" (DISCLAIMER: I have not encountered this idea anywhere in DL, that's why I have taken the liberty of naming it, and I don't provide citation. If anyone knows if that's a thing, please reach out to me with the relevant sources!).

I'm not going to delve into that in this post. Keep an eye out for that, though, because a detailed approach to explaining this idea, as well as its potential benefits to DL models, will be posted soon.

Interpretation

interp = ClassificationInterpretation.from_learner(learn)
interp.plot_confusion_matrix()
interp.plot_top_losses(41, nrows=10)
target predicted probabilities loss
0 Shield;Surgical Cloth;Medical tensor([0.0077, 0.9984, 0.0078, 0.2098, 0.9549, 0.0643, 0.0777, 0.0062]) 2.187938690185547
1 Cloth Cloth;Medical tensor([0.0247, 0.9129, 0.0176, 0.3189, 0.9997, 0.0585, 0.0198, 0.0138]) 1.072890043258667
2 KN95 Cloth tensor([0.0213, 0.9823, 0.0130, 0.0497, 0.1453, 0.1231, 0.2686, 0.1380]) 0.9774149656295776
3 Cloth Cloth;KN95;Shield tensor([0.0695, 0.5541, 0.0023, 0.8898, 0.0520, 0.0520, 0.9902, 0.0553]) 0.9573626518249512
4 KN95 Cloth tensor([0.1523, 0.9844, 0.0093, 0.1110, 0.0942, 0.0725, 0.0575, 0.0278]) 0.8492633104324341
5 Cloth;Medical - tensor([0.9499, 0.3249, 0.0056, 0.0223, 0.3289, 0.0488, 0.0469, 0.0617]) 0.6774997711181641
6 Medical KN95;Surgical tensor([0.0414, 0.0395, 0.0103, 0.8503, 0.1052, 0.0268, 0.0749, 0.6215]) 0.6650393009185791
7 Cloth;KN95;Medical Cloth;Medical tensor([0.0505, 0.9252, 0.0441, 0.0282, 0.7488, 0.0972, 0.0126, 0.0173]) 0.5204622149467468
8 Shield;Surgical Surgical tensor([0.0435, 0.0351, 0.0060, 0.0195, 0.1074, 0.0586, 0.0225, 1.0000]) 0.5091105103492737
9 Gas Gas;Shield tensor([0.0169, 0.1474, 0.9952, 0.0050, 0.1380, 0.1350, 0.9696, 0.0539]) 0.5035613179206848
10 Medical Cloth tensor([0.0402, 0.9010, 0.0150, 0.1106, 0.2715, 0.0434, 0.0132, 0.0726]) 0.49044445157051086
11 Cloth KN95;Medical tensor([0.0313, 0.3274, 0.0054, 0.8075, 0.5183, 0.0214, 0.0184, 0.0662]) 0.4551154375076294
12 KN95 Cloth tensor([0.0629, 0.7515, 0.0389, 0.1632, 0.1261, 0.0354, 0.0563, 0.0210]) 0.44490036368370056
13 Medical tensor([0.1121, 0.3474, 0.0093, 0.2354, 0.0976, 0.0687, 0.0074, 0.2106]) 0.4331527352333069
14 Cloth;Medical;Shield Cloth;Medical tensor([0.0250, 0.9994, 0.0053, 0.0496, 0.8723, 0.0696, 0.0463, 0.0132]) 0.42210978269577026
15 Medical Cloth tensor([0.0294, 0.7738, 0.0484, 0.0403, 0.2920, 0.1259, 0.1379, 0.0157]) 0.3921005129814148
16 Cloth;KN95;Medical Cloth;Medical tensor([0.0176, 0.5974, 0.0060, 0.0978, 1.0000, 0.0401, 0.0211, 0.0606]) 0.3735991418361664
17 KN95 KN95;Shield tensor([0.0200, 0.1745, 0.0029, 0.9935, 0.1081, 0.0765, 0.8843, 0.2139]) 0.35156717896461487
18 Shield KN95;Shield tensor([0.0181, 0.0419, 0.0024, 0.9099, 0.0988, 0.0316, 0.9965, 0.0137]) 0.327982634305954
19 Surgical - tensor([0.7569, 0.0316, 0.0045, 0.0167, 0.1484, 0.0656, 0.0061, 0.4032]) 0.32634592056274414
20 Surgical tensor([0.0282, 0.4589, 0.0152, 0.0391, 0.1623, 0.1415, 0.1643, 0.2511]) 0.32359299063682556
21 No_mask! Cloth;No_mask! tensor([0.3992, 0.7288, 0.0027, 0.0110, 0.1893, 0.8107, 0.0027, 0.1024]) 0.2948143184185028
22 Cloth Cloth;Surgical tensor([0.0578, 0.6261, 0.0037, 0.0493, 0.1708, 0.0303, 0.1064, 0.6651]) 0.25082236528396606
23 Cloth tensor([0.0534, 0.4188, 0.0059, 0.4405, 0.1227, 0.1601, 0.0911, 0.0557]) 0.24625501036643982
24 Cloth tensor([0.2052, 0.2710, 0.0556, 0.0713, 0.1052, 0.0408, 0.0304, 0.0265]) 0.23463144898414612
25 No_mask! -;No_mask! tensor([0.5985, 0.3772, 0.0043, 0.0417, 0.2019, 0.9302, 0.0539, 0.0099]) 0.22453458607196808
26 Cloth Cloth;Surgical tensor([0.0708, 0.9986, 0.0024, 0.0011, 0.2059, 0.0593, 0.0077, 0.7262]) 0.2091251015663147
27 Surgical Shield;Surgical tensor([0.0698, 0.0179, 0.0041, 0.0595, 0.1548, 0.0284, 0.7116, 1.0000]) 0.19954973459243774
28 Cloth Cloth;Medical tensor([0.1025, 0.9139, 0.0071, 0.0101, 0.6403, 0.0783, 0.0440, 0.0036]) 0.17099031805992126
29 KN95 KN95 tensor([0.2454, 0.1503, 0.0077, 0.8222, 0.2953, 0.0441, 0.1083, 0.0736]) 0.15426596999168396
30 Shield;Surgical Shield tensor([0.0613, 0.0185, 0.0017, 0.0955, 0.1229, 0.0527, 1.0000, 0.4334]) 0.15068519115447998
31 Shield;Surgical Shield tensor([0.0613, 0.0185, 0.0017, 0.0955, 0.1229, 0.0527, 1.0000, 0.4334]) 0.15068519115447998
32 Cloth Cloth tensor([0.0271, 0.9734, 0.0240, 0.1700, 0.4952, 0.0207, 0.0316, 0.1311]) 0.14279848337173462
33 - - tensor([0.9376, 0.0965, 0.4565, 0.0216, 0.0858, 0.0359, 0.0380, 0.0883]) 0.13187271356582642
34 Surgical Surgical tensor([0.0828, 0.3826, 0.0157, 0.0843, 0.1102, 0.0422, 0.1392, 0.9757]) 0.1258525252342224
35 Cloth Cloth;Medical tensor([0.0189, 1.0000, 0.0043, 0.0043, 0.5717, 0.0649, 0.0155, 0.0093]) 0.12096662819385529
36 Cloth Cloth tensor([0.1844, 0.9992, 0.0032, 0.0060, 0.2830, 0.0134, 0.2912, 0.0589]) 0.12058921158313751
37 - - tensor([0.9853, 0.3103, 0.0036, 0.0394, 0.1954, 0.1406, 0.0307, 0.0200]) 0.10630811750888824
38 Cloth Cloth tensor([0.1158, 0.9534, 0.0564, 0.1340, 0.1559, 0.0955, 0.1095, 0.0608]) 0.10266871750354767
39 - -;Medical tensor([0.9997, 0.0311, 0.0061, 0.0070, 0.5063, 0.0262, 0.0277, 0.0118]) 0.10216984152793884
40 Surgical Surgical tensor([0.4630, 0.0049, 0.0096, 0.0046, 0.0630, 0.0538, 0.0532, 1.0000]) 0.10198771208524704

After interpreting the results (for some wicked reason I can't seem to get the "Confusion matrix" to work with my ds) by viewing the top losses, I realize that the model misses a few easy ones, all the while being extremely accurate at useless datapoints (more on that later).

  • It should be noted that I DID NOT train the first model until overfitting. I stopped mainly because I had ideas I deemed more valuable to explore, and because of the threshold to accuracy relationship, which suggested that the model is almost as good as possible. Nowadays, I tend to dismiss such early stoppings, given that these ploted relationships are but predictions.

Second Try

Firstly, I cleaned the ds, as I thought I had done earlier. I merged the "Surgical" and "Medical" categories into "Medical", and relabeled a few misclassified items.

In continuation, I will tweak slightly the hyperparameters to try to achieve better performance than the above reached (~95%), including cropping to a bigger size (224 to start with) and using Presizing, using a deeper arch, and adjusting the threshold/lr.

Note: I deleated the 224 crop results accidentally during training. It was slightly better than the above reached accuracy, even though only by a percentage point (~96%). I have included, though, below the next stage; applying Presizing with far higher resolution in the hopes of achieving near perfect accuracy.

Data

df = pd.read_csv('CSVs/only_images4.csv')
df
fname labels
0 C001.jpg Cloth
1 C002.jpg Cloth
2 C003.jpg Cloth
3 C004.jpg Cloth
4 C005.jpg Cloth
... ... ...
1009 No094.jpg No_mask!
1010 No096.jpg No_mask!
1011 No097.jpg No_mask!
1012 No099.jpg No_mask!
1013 No100.jpg No_mask!

1014 rows × 2 columns

# Here is the main difference of this time; use of Presizing (720->460), in the hopes of higher accuracy due to higher resolution 

dblock = DataBlock(blocks=(ImageBlock, MultiCategoryBlock),
                   splitter=RandomSplitter(seed=42),
                   get_x=get_x, 
                   get_y=get_y,
                   item_tfms=Resize(720),
                   
                   batch_tfms=aug_transforms(size=460, min_scale=0.75))

dsets = dblock.datasets(df)

# I got a CUDA memory error; that's why the batch size had to be smaller than 64, and I traditionally chose 32
dls = dblock.dataloaders(df, bs=32)
dls.show_batch()

Training

# Threshold = 0.8 (instead of 0.2)
learn0 = cnn_learner(dls, resnet50, metrics=partial(accuracy_multi, thresh=0.8))

I tried to first get the lr_min, in order to better direct the training. Pretty much expected curve

lr_min, lr_steep = learn0.lr_find()

Cuda memory error

t = torch.cuda.get_device_properties(0).total_memory
c = torch.cuda.memory_cached(0)
a = torch.cuda.memory_allocated(0)
# f = c-a  # free inside cache
t, c, a
/opt/conda/envs/fastai/lib/python3.8/site-packages/torch/cuda/memory.py:344: FutureWarning: torch.cuda.memory_cached has been renamed to torch.cuda.memory_reserved
  warnings.warn(
(17069309952, 14545846272, 112194560)

First set of epochs

learn0.fine_tune(3, base_lr=lr_min, freeze_epochs=3)
epoch train_loss valid_loss accuracy_multi time
0 0.707048 0.434021 0.922207 00:27
1 0.450835 0.140279 0.944130 00:27
2 0.285841 0.141349 0.962518 00:28
epoch train_loss valid_loss accuracy_multi time
0 0.218881 0.932779 0.888968 00:37
1 0.220071 0.265597 0.937058 00:37
2 0.172344 0.096667 0.958274 00:37
lr_min, lr_st = learn0.lr_find()

Second set of epochs

learn0.fine_tune(2, base_lr=lr_min, freeze_epochs=3)
epoch train_loss valid_loss accuracy_multi time
0 0.101329 0.086775 0.960396 00:28
1 0.095762 0.086316 0.959689 00:28
2 0.089909 0.086750 0.959689 00:28
epoch train_loss valid_loss accuracy_multi time
0 0.095427 0.088182 0.959689 00:37
1 0.091889 0.085616 0.961103 00:37
preds,targs = learn0.get_preds()
xs = torch.linspace(0.01,0.99,350)
accs = [accuracy_multi(preds, targs, thresh=i, sigmoid=False) for i in xs]
plt.plot(xs,accs);

Third and final set of epochs

learn0.fine_tune(2, base_lr=lr_min, freeze_epochs=2)
epoch train_loss valid_loss accuracy_multi time
0 0.097029 0.089718 0.958982 00:28
epoch train_loss valid_loss accuracy_multi time
0 0.094402 0.089689 0.960396 00:37
1 0.093266 0.086641 0.961811 00:37
preds,targs = learn0.get_preds()
xs = torch.linspace(0.05,0.95,350)
accs = [accuracy_multi(preds, targs, thresh=i, sigmoid=False) for i in xs]
plt.plot(xs,accs);

I increase the number of points to search from up to 350. The difference in plot shapes is obvious, albeit of limited practical importance.

Interpretation

interp = ClassificationInterpretation.from_learner(learn0)
interp.plot_confusion_matrix()
interp.plot_top_losses(40, nrows=10)
target predicted probabilities loss
0 - Medical tensor([7.1864e-03, 1.9517e-02, 5.7940e-03, 6.6158e-03, 9.1515e-01, 6.4330e-04, 7.1826e-02]) 1.0728305578231812
1 Medical;Shield - tensor([9.4860e-01, 8.2339e-02, 7.8895e-03, 2.3476e-02, 1.2792e-01, 1.5864e-04, 1.0363e-01]) 1.0584402084350586
2 KN95;Shield Cloth tensor([0.0024, 0.8134, 0.0098, 0.1013, 0.1418, 0.0074, 0.1106]) 0.9061436653137207
3 Cloth Gas tensor([1.4924e-02, 3.0396e-02, 8.8559e-01, 8.5396e-02, 5.0591e-05, 5.4732e-04, 8.9289e-04]) 0.8238886594772339
4 No_mask! - tensor([0.7779, 0.0738, 0.0020, 0.0144, 0.0422, 0.0209, 0.0432]) 0.7935909628868103
5 KN95;Shield Shield tensor([0.3309, 0.0627, 0.0055, 0.0164, 0.1653, 0.0027, 0.6454]) 0.7432813048362732
6 Cloth;KN95;Medical Cloth tensor([1.7814e-04, 9.5576e-01, 3.8822e-05, 1.8135e-02, 3.3463e-01, 2.3627e-03, 9.8615e-04]) 0.7362056970596313
7 Gas - tensor([0.7025, 0.0204, 0.0410, 0.0281, 0.0105, 0.0157, 0.0047]) 0.6411075592041016
8 KN95;Shield Cloth;Shield tensor([0.0148, 0.5299, 0.0042, 0.0596, 0.0646, 0.0011, 0.7471]) 0.5646962523460388
9 Cloth;KN95 Cloth;Medical tensor([0.0164, 0.7377, 0.0166, 0.0930, 0.6964, 0.0030, 0.0013]) 0.5584717988967896
10 Medical;Shield Medical tensor([7.7260e-05, 2.4786e-04, 2.5288e-06, 1.0900e-05, 9.9973e-01, 1.1516e-06, 2.5888e-02]) 0.5220851898193359
11 Medical;Shield Medical tensor([8.0212e-05, 1.3874e-04, 4.7152e-06, 6.7401e-06, 9.9989e-01, 1.2644e-06, 3.4239e-02]) 0.48210400342941284
12 Cloth tensor([1.8111e-04, 6.7206e-02, 2.0173e-03, 1.7350e-02, 4.4227e-01, 1.1182e-03, 5.5856e-02]) 0.4803086221218109
13 Medical Cloth;Medical tensor([6.0449e-03, 9.4671e-01, 9.7263e-05, 4.8402e-02, 8.8149e-01, 6.1641e-04, 5.9388e-03]) 0.4457746148109436
14 Cloth;Shield Shield tensor([0.0735, 0.1129, 0.0115, 0.0598, 0.1168, 0.1199, 0.6379]) 0.4332104027271271
15 - tensor([1.4689e-01, 4.0333e-01, 3.8488e-04, 2.5451e-02, 1.7683e-02, 3.5612e-01, 2.9692e-03]) 0.4173838198184967
16 Cloth;Shield Cloth tensor([5.3234e-04, 8.4760e-01, 3.0976e-03, 5.6098e-02, 3.5594e-01, 1.9782e-03, 1.0929e-01]) 0.4117661714553833
17 Medical Shield tensor([3.6909e-04, 5.1937e-03, 6.3702e-03, 3.9115e-02, 4.6684e-01, 1.4459e-03, 8.4203e-01]) 0.38006311655044556
18 Medical Medical;Shield tensor([1.4350e-03, 5.0586e-03, 9.7197e-04, 2.2856e-03, 9.0900e-01, 3.1733e-04, 9.1633e-01]) 0.3694818615913391
19 - Cloth tensor([0.3624, 0.7249, 0.0532, 0.0152, 0.0029, 0.0012, 0.1324]) 0.36024340987205505
20 Medical tensor([1.0746e-04, 1.6511e-02, 8.3923e-04, 3.2906e-01, 1.5616e-01, 2.8831e-04, 1.9375e-02]) 0.32763299345970154
21 Shield tensor([0.0384, 0.1765, 0.0033, 0.0331, 0.1166, 0.1871, 0.2049]) 0.3123726546764374
22 Gas Cloth tensor([2.6330e-03, 5.6460e-01, 2.9816e-01, 2.8259e-02, 5.7453e-03, 5.7981e-03, 1.5344e-05]) 0.29778796434402466
23 Shield No_mask!;Shield tensor([1.8141e-03, 8.6844e-03, 2.3820e-04, 3.2092e-02, 8.4428e-03, 7.2501e-01, 6.3948e-01]) 0.25571224093437195
24 Medical;Shield Medical tensor([6.6729e-05, 2.0053e-05, 2.2359e-06, 5.1963e-06, 9.9997e-01, 5.4061e-07, 1.7936e-01]) 0.24549484252929688
25 Gas tensor([0.0445, 0.4098, 0.4082, 0.0930, 0.0095, 0.0127, 0.0837]) 0.239467591047287
26 Medical;Shield Medical tensor([4.8403e-05, 1.8774e-04, 1.6524e-05, 2.7448e-05, 9.9934e-01, 5.4019e-06, 1.8736e-01]) 0.2393771857023239
27 Medical;Shield Medical tensor([1.2438e-04, 1.0014e-05, 4.7314e-06, 3.9429e-06, 9.9997e-01, 1.1959e-06, 1.9395e-01]) 0.2343320995569229
28 Cloth Cloth;Medical tensor([1.5245e-02, 9.7492e-01, 7.8498e-05, 8.6758e-02, 7.7415e-01, 1.3900e-04, 2.3870e-04]) 0.23140935599803925
29 No_mask! tensor([0.3722, 0.0604, 0.0008, 0.0114, 0.0044, 0.3931, 0.0162]) 0.21351192891597748
30 Medical;Shield Medical tensor([7.7823e-05, 3.0472e-04, 7.8055e-05, 2.0316e-04, 9.9646e-01, 1.2870e-05, 2.2650e-01]) 0.2127458155155182
31 Medical;Shield Medical tensor([1.5801e-04, 2.7898e-04, 3.7677e-05, 3.9934e-05, 9.9944e-01, 8.0454e-06, 2.5164e-01]) 0.1972629874944687
32 Shield Cloth;Shield tensor([8.5926e-03, 6.1524e-01, 5.3632e-03, 2.3268e-02, 8.1071e-03, 5.4811e-04, 6.8703e-01]) 0.19667738676071167
33 KN95 tensor([0.0018, 0.0974, 0.0242, 0.3349, 0.0514, 0.0027, 0.0128]) 0.1844436228275299
34 Cloth;Medical Cloth tensor([1.8994e-04, 9.5007e-01, 2.0903e-04, 3.6779e-02, 3.4507e-01, 2.3358e-02, 6.6488e-04]) 0.16820193827152252
35 Shield Shield tensor([0.1247, 0.2386, 0.0015, 0.0478, 0.0223, 0.0024, 0.5072]) 0.16572348773479462
36 Medical;Shield Shield tensor([5.2771e-04, 3.7455e-02, 2.6122e-02, 1.2338e-02, 3.7254e-01, 1.2216e-03, 9.6325e-01]) 0.15766814351081848
37 Cloth tensor([3.1620e-04, 4.4886e-01, 3.6443e-03, 3.3457e-02, 3.9798e-03, 1.5899e-01, 2.9744e-03]) 0.14559409022331238
38 Gas Gas tensor([0.0038, 0.0341, 0.6053, 0.0717, 0.0194, 0.0156, 0.2760]) 0.13902735710144043
39 No_mask! No_mask! tensor([0.0397, 0.3198, 0.0012, 0.0885, 0.0116, 0.7440, 0.1094]) 0.13470366597175598

Manual Testing

learn0.predict('WhatsApp Image 2020-09-13 at 8.34.08 PM.jpeg')
((#1) ['KN95'],
 tensor([False, False, False,  True, False, False, False]),
 tensor([3.7567e-04, 1.4866e-02, 1.9021e-03, 9.3934e-01, 1.2692e-03, 6.1743e-02, 1.6483e-03]))
learn0.predict('WhatsApp Image 2020-09-13 at 8.34.23 PM.jpeg')
((#1) ['KN95'],
 tensor([False, False, False,  True, False, False, False]),
 tensor([1.9798e-04, 2.5388e-03, 7.8868e-04, 9.8569e-01, 4.1836e-03, 1.0470e-03, 1.8771e-03]))
learn0.predict('family.jpg')
((#0) [],
 tensor([False, False, False, False, False, False, False]),
 tensor([2.4998e-04, 4.1010e-03, 6.8737e-03, 3.9971e-01, 6.2634e-04, 3.4899e-01, 1.5314e-02]))
learn0.predict('xr.png')
((#1) ['Medical'],
 tensor([False, False, False, False,  True, False, False]),
 tensor([7.4280e-04, 1.4337e-03, 3.9989e-04, 7.5603e-04, 9.9519e-01, 3.1935e-05, 6.4606e-02]))
learn0.predict('boom.PNG')
((#0) [],
 tensor([False, False, False, False, False, False, False]),
 tensor([0.3424, 0.0293, 0.0019, 0.0444, 0.0084, 0.0394, 0.0300]))
learn0.predict('t.PNG')
((#1) ['No_mask!'],
 tensor([False, False, False, False, False,  True, False]),
 tensor([0.0467, 0.0834, 0.0065, 0.1329, 0.0323, 0.5063, 0.0385]))
!pwd
/notebooks/Mannual/EXTRA NEW

Review - Second Try

The model did excellent in the Manual Testing phase. It did not make any wrong predictions. However, these results and the top loses propelled me to understand what is really happening; when you increase the resolution, the model gets more and more inaccurate at recognizing the types of what it is searching for, and their characteristic differences.

That means that as the model gets better at figuring out how to not classify an image that does not contain a mask as a type of a mask it gets worse at locating the differencies between types of masks. The consequencies are that the model is more accurate and confident about things not being, rather than them being, what it is searching for.

# As a clarification, these are the possible categories for the time being that the model has been trained to recognize
learn0.dls.vocab
(#7) ['-','Cloth','Gas','KN95','Medical','No_mask!','Shield']

Anyways, it's a fairly good model, with potential practical use. That's why I decided to export it.

learn0.export('masksv3_modelv01.pkl')

Conclusion

In this post, I covered as well as I could the process which I followed to create the "masksv3_modelv01.pkl" model.

The journey, however, is far from over. To achieve the model displayed at https://protected-mountain-28715.herokuapp.com/ as a free, available for anyone to use application, I had to recreate the dataset a few more times, as well as to explore distinct approaches and peculiar ideas (out of which the most important could be "Intended Noise"), among other things.

Make sure to get in touch with me for any suggestions, comments or corrections. A respectable and thoughtful comment is always welcome from anyone.